feat(dataset): add `collect_all_keys` option and keep CSV export columns aligned by anxkhn · Pull Request #2079 · apify/crawlee-python

anxkhn · 2026-07-16T07:17:30Z

Fixes CSV export writing values under the wrong column. Rows were written positionally with
csv.writer, so any item whose key set or key order differed from the first item's had its values
shifted or placed under the wrong header. Rows are now written by key with csv.DictWriter.
Columns are taken from the first non-empty item. Keys that appear only in later items are dropped
instead of being appended out of position, and an item missing a column now gets an empty cell
instead of a short row. Exports of datasets with mixed keys therefore change content.
Dropped keys are reported once per export in a warning that names them. Crawlee for JS drops them
silently, so this is a Python-only addition.
Adds a collect_all_keys parameter to Dataset.export_to and BasicCrawler.export_data. When
True, the columns are the union of all item keys in first-seen order, so nothing is dropped. It
reads the whole dataset before writing the header, unlike the default. Named and behaving like
collectAllKeys in Crawlee for JS.
Adds restval, the value written for columns an item has no key for. It was rejected before,
because csv.writer has no such argument.
Documents ExportDataCsvKwargs as mirroring csv.DictWriter and ExportDataJsonKwargs as
mirroring json.dump, so Crawlee-specific options stay out of them.
Documents the column behavior in the dataset export example.
Alignment with Crawlee for JS: the by-key row mapping, deriving columns from the first item, and
collectAllKeys all match Dataset.exportTo in packages/core/src/storages/dataset.ts.

✍️ Drafted by Claude Code

Mantisus

Thank you for finding this bug. However, we cannot use extrasaction=‘ignore’

Copilot

Pull request overview

This PR fixes CSV export for Dataset items that don’t share identical key order/sets, preventing values from being written under the wrong column when exporting heterogeneous items.

Changes:

Update export_csv_to_stream to write rows by key (via csv.DictWriter) instead of positional item.values().
Add regression tests covering heterogeneous item keys/order and skipping empty items during header selection.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
`src/crawlee/_utils/file.py`	Switches CSV export implementation to key-based writing and changes how CSV headers are derived.
`tests/unit/_utils/test_file.py`	Adds regression tests for column alignment with heterogeneous items and for skipping empty mappings.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

`export_csv_to_stream` built the header from the first item's keys but then wrote every row positionally with `csv.writer` and `item.values()`. Since `Dataset.push_data` accepts arbitrary mappings per item with no schema, items whose keys differ in membership or order from the first item had their values written under the wrong columns (or shifted into columns that do not exist), silently corrupting the CSV export of any non-uniform dataset. Write rows with `csv.DictWriter` keyed on the first item's fields and `extrasaction='ignore'`, so each value lands in its own column and unexpected keys are dropped. This matches the column-aligned behavior of Crawlee for JavaScript. Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>

anxkhn · 2026-07-27T12:56:29Z

pushed a revision addressing the latest review and ci feedback.

vdusek · 2026-07-27T13:13:20Z

Same as here #2074 (comment)

@anxkhn Thank you. For future reviews, I'd really appreciate it if you could address all the feedback in a single follow-up commit, rather than splitting the changes across multiple commits, merging them into earlier commits, and force-pushing it. Your current approach makes the changes really hard to review.

vdusek

A few important things and then a few nits.

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>

anxkhn · 2026-07-27T16:53:11Z

@vdusek addressed all new threads, including js-default parity, streaming behavior, bounded default memory, and value preservation tests. rebased and re-requesting review. thank you.

…eagerly Address review findings on the CSV export rewrite: - Warn once per export, naming every key dropped because it is absent from the first item, instead of losing those values silently. Master wrote them to the wrong column, which was wrong but at least visible. - Build a throwaway writer up front so invalid options (e.g. `delimiter='ab'`) raise even when the dataset is empty. Master validated eagerly; the lazy writer made a misconfigured export succeed on a run that scraped nothing and fail on the next one. - Pass `extrasaction='ignore'` on the buffered path too, so both paths construct the writer identically and narrowing `fieldnames` later cannot raise mid-export. - Declare `restval`, which already worked but was rejected by the type checker at every call site, and correct the `ExportDataCsvWriterKwargs` docstring that named `csv.DictWriter` while listing only `csv.writer` options. - Cover the empty-items guard, the `if item` filter on the buffered path, and `collect_all_keys` through `Dataset.export_to` and `BasicCrawler.export_data`, none of which had a test. - Document the column behavior and `collect_all_keys` in the dataset export example.

vdusek · 2026-07-28T08:42:10Z

Summary:

Fixes CSV export writing values under the wrong column. Rows were written positionally with
csv.writer, so any item whose key set or key order differed from the first item's had its values
shifted or placed under the wrong header. Rows are now written by key with csv.DictWriter.
Columns are taken from the first non-empty item. Keys that appear only in later items are dropped
instead of being appended out of position, and an item missing a column now gets an empty cell
instead of a short row. Exports of datasets with mixed keys therefore change content.
Dropped keys are reported once per export in a warning that names them. Crawlee for JS drops them
silently, so this is a Python-only addition.
Adds a collect_all_keys parameter to Dataset.export_to and BasicCrawler.export_data. When
True, the columns are the union of all item keys in first-seen order, so nothing is dropped. It
reads the whole dataset before writing the header, unlike the default. Named and behaving like
collectAllKeys in Crawlee for JS.
Adds restval, the value written for columns an item has no key for. It was rejected before,
because csv.writer has no such argument.
Documents ExportDataCsvKwargs as mirroring csv.DictWriter and ExportDataJsonKwargs as
mirroring json.dump, so Crawlee-specific options stay out of them.
Documents the column behavior in the dataset export example.
Alignment with Crawlee for JS: the by-key row mapping, deriving columns from the first item, and
collectAllKeys all match Dataset.exportTo in packages/core/src/storages/dataset.ts.

✍️ Drafted by Claude Code

Mantisus

LGTM

…hed kwargs

…mn-alignment

vdusek

LGTM

Mantisus suggested changes Jul 16, 2026

View reviewed changes

Comment thread src/crawlee/_utils/file.py Outdated

Comment thread tests/unit/_utils/test_file.py

vdusek requested review from Copilot and vdusek July 17, 2026 14:18

Copilot started reviewing on behalf of vdusek July 17, 2026 14:18 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

Comment thread src/crawlee/_utils/file.py Outdated

Comment thread src/crawlee/_utils/file.py Outdated

vdusek requested changes Jul 20, 2026

View reviewed changes

Comment thread src/crawlee/_utils/file.py Outdated

Comment thread src/crawlee/_utils/file.py Outdated

anxkhn added 3 commits July 27, 2026 18:26

fix(dataset): include all CSV export columns

6e60ba7

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>

fix(dataset): bound CSV export memory usage

bde4ba2

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>

anxkhn force-pushed the fix/csv-export-column-alignment branch from 5b95959 to bde4ba2 Compare July 27, 2026 12:56

anxkhn changed the title ~~fix(dataset): keep CSV columns aligned for items with different keys~~ fix(dataset): bound CSV export memory usage Jul 27, 2026

vdusek reviewed Jul 27, 2026

View reviewed changes

fix(dataset): stream CSV exports by default

35ee524

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>

vdusek changed the title ~~fix(dataset): bound CSV export memory usage~~ fix(dataset): keep CSV export columns aligned Jul 28, 2026

test(dataset): decouple CSV writer option assertion from CPython wording

94ba83e

Mantisus approved these changes Jul 28, 2026

View reviewed changes

vdusek changed the title ~~fix(dataset): keep CSV export columns aligned~~ feat(dataset): add collect_all_keys option and keep CSV export columns aligned Jul 28, 2026

vdusek added 4 commits July 28, 2026 12:29

fix(dataset): avoid CSV export crashes on non-string keys and mismatc…

d090f55

…hed kwargs

Merge remote-tracking branch 'origin/master' into fix/csv-export-colu…

ef2eb49

…mn-alignment

fix(dataset): warn about export_data kwargs of the other format

46da14f

fix(dataset): stop filtering export_data kwargs by format

a7f475d

vdusek changed the title ~~feat(dataset): add collect_all_keys option and keep CSV export columns aligned~~ feat(dataset): add collect_all_keys option and keep CSV export columns aligned Jul 28, 2026

refactor(dataset): make collect_all_keys an explicit export parameter

0384d3f

vdusek changed the title ~~feat(dataset): add collect_all_keys option and keep CSV export columns aligned~~ feat(dataset): add collect_all_keys option and keep CSV export columns aligned Jul 28, 2026

vdusek approved these changes Jul 28, 2026

View reviewed changes

vdusek merged commit cd1e1f6 into apify:master Jul 28, 2026
34 checks passed

Uh oh!

Conversation

anxkhn commented Jul 16, 2026 • edited by vdusek Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Mantisus left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

anxkhn commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vdusek commented Jul 27, 2026

Uh oh!

vdusek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

anxkhn commented Jul 27, 2026

Uh oh!

vdusek commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Mantisus left a comment

Choose a reason for hiding this comment

Uh oh!

vdusek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

anxkhn commented Jul 16, 2026 •

edited by vdusek

Loading

anxkhn commented Jul 27, 2026 •

edited

Loading

vdusek commented Jul 28, 2026 •

edited

Loading